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ABSTRACT 

The  Marine  Corps  Combat  Readiness  Evaluation  System 
(MCCRE3)  was  designed  to  provide  timely  and  accurate  infor- 
mation concerning  the  ability  of  active  and  reserve  forces 
to   carry      out  assigned      combat    missions-  To   provide      this 

information,  units  are  subjected  to  simulated  combat  prob- 
lems and  their  performance  is  observed  by  expert  evaluators 
from  within  the  Marine  Corps.  Though  these  evaluators  are 
considered  experts  in  their  fields,  they  may  inject  bias 
into  their  evaluations  causing  an  inaccurate  combat  readi- 
ness   rating   for    the    unit    observed. 

Analysis  of  the  MCCHES  reveals  three  main  areas  where 
evaluatcr      bias    may      appear:  senior   evaluator      influence, 

ether  evaluator  bias  and  interpretation  of  the  mission 
performance  standards  used  tc  conduct  the  evaluation.  To 
alleviate  these  problems,  three  actions  are  explored:  evalu- 
ator training,  evaluator  testing  and  quantification  of  the 
mission    performance    standards. 
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I.    IN3RQDUCTI0N 

A-       PURPOSE 

The  purpose  cf  this  paper  is  to  examine  the  Marine  Corps 
Combat  Readiness  Evaluation  System  (MCCRES)  to  discover  if 
the  system  is  susceptible  to  biases  which  may  cause  the 
results  cf  evaluations  to  inaccurately  reflect  the  ccmbat 
readiness  of  evaluated  units.  To  guide  research,  two 
specific   questions    are   posed: 

1.  Can   factors   of    the      MCCRES    evaluation    which   are 
subject    to    evaiuator   bias   be    identified? 

2.  How        can      these      factors      be        controlled     or 
controlled    for? 

E.       BACKGROUND 

The  Marine  Corps  Combat  Readiness  Evaluation  System  was 
designed  tc  provide  timely  and  accurate  information 
concerning  the  ability  of  operating  units  of  the  Marine 
Corps,  bcth  active  and  reserve,  tc  carry  out  assigned  ccmbat 
missions.  The  system  uses  "expert"  evaluators  from  various 
specialty  areas  to  observe  and  grade  simulated  combat  opera- 
tions. Aggregating  these  evaluations  provides  an  overall 
view  cf  a  unit's  readiness  for  combat,  and  feedback  from  the 
evaluation  allows  the  unit  commander  to  identify  and  correct 
potentially    problematic   areas   within    his   command. 

Though  the  MCCRES  is  relied  upon  as  a  standard  against 
which  units  are  judged,  the  readiness  grade  received  could 
be  mere  dependent  upen  the  evaiuator  than  the  actual  task 
performance  being  graded.  By  controlling  cr  controlling  for 
evaluatcr  bias,  a  mere  uniform  standard  by  which  to  judge 
ccmbat   readiness  can    te   realized. 
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C.       SCOPE    AND   METHODOLOGY 

This  thesis  viass  tha  MCCRES  as  an  information  system 
and  explores  areas  where  e valuator  bias  (input)  can  cause 
ratings  (output)  to  reflect  the  evaluator's  opinion  rather 
than  the  mission  performance  of  the  evaluated  unit.  Two 
major   topics  are  researched: 

1.  Evaluation — Its    major   approaches    and   principles 

2.  Evaluators — Their   sources    and   typical    errors 
These      areas     are      related      to   the      MCCRES      and      methods     of 
controlling   or    controlling    for    evaluator   bias   are    developed. 

The  research  consists  of  a  detailed  literature  search  in 
the  area  of  evaluation  science.  Methods  for  the  reduction  or 
ccntrcl  of  evaluator  bias  are  explored  for  use  in  the 
context    cf    the    MCCRES. 
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II-     EVALUATION 

This  chapter  addresses  the  evaluation  process, 
presenting  definitions,  purposes  and  principles  of  evalua- 
tion, and  explores  some  currently  used  approaches  for 
conducting  evaluations.  The  questions  of  what  to  evaluate 
and   when   to   evaluate   are   also   investigated. 

The  terms  goal  and  objective  are  used  throughout  this 
and  succeeding  chapters.  Objectives  refer  to  long  range 
statements  cf  purpose  within  the  organization.  They  gener- 
ally can  not  be  specifically  stated  and  need  not  be  attain- 
able in  the  immediate  future.  Alternatively,  aoais  are  more 
readily  attainable  in  the  short  run  and  are  specifically 
stated.  They  can  appear  as  written  statements  which  guide  an 
organizaticn 's  operations,  and  are  a  standard  against  which 
performance   can    be    measured. 

A.       DEFINITION     AND    PURPOSE    OF    EVALUATION 

1  •      S e f i ni t i en    of   Evaluation 

There  are  many  definintions  of  the  term  evaluation. 
Bather  than  select  a  single  author's  definition,  two  obser- 
vations and  two  definitions  of  evaluation  are  presented  here 
to  shew  hoth  the  similarities  and  differences  encountered  in 
the  field  cf  evaluation  research.  These  definitions  and 
observations  are  given  in   order   from    simple   to   rigorcus. 

The  first,  mere  an  observation  than  a  definition,  is 
from    E.R.    House: 


At  its  simplest,  evaluation  leads  to  a  settled  opinion 
that  something  is  the  case.  It  does  not  necessarily  lead 
to  a  decision  to  act  in  a  certain  way,  though  toaay  it 
is  cften  intended  for  that  purpose. ...  Evaluation  leads 
to  a  judgement  about  the  worth  of  something. 
[Ref.    1:p.18] 
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The  second  observation  about  evaluation,  in  partic- 
ular the  evaluation  of  a  process,  is  that  its  scope  "is 
confined  to  assessing  what  a  particular  program  has  accom- 
plished in  meeting  its  immediate  objectives...,"  and 
assessing   the  "workability    "   of   a   program   [Ref.    2    :p.11]. 

Henry  W.  Rieken's  definition  looks  upon  evaluation 
as  "  ths  measurement  of  desireable  and  undesireable  conse- 
quences of  an  action  that  has  been  taken  in  order  to  forward 
some    goal   that    we  value."      [Ref.    3   :p.54] 

Finally,  ths  definition  presented  by  Stufflebeam  et 
al. ,  is  that  "...evaluation  is  the  process  cf  delineating, 
obtaining,  and  providing  useful  information  for  judging 
decision   alternatives."     [Ref.    4    :p.40] 

There  are  two  factors  common  to  each  of  the 
preceeding  observations  and  definitions.  First,  evaluation 
is  concerned  with  making  a  judgement  or  assessment  about 
something.  Second,  that  judgement  can  be  made  in  terms  of 
some  goal  or  objective.  These  two  factors  are  used  as  a 
basis  for  a  definition  of  evaluation  developed  in  the  next 
section. 

2 •      Purgcse    of    E valuati  on 

Using  the  atcve  descriptions  of  evaluation,  the 
purpose  of  evaluation  can  be  examined.  Stufflebeam  et  al.  , 
stated  simply  that  "The  purpose  of  evaluation  is  not  to 
prove  but  to  improve."  [Ref.  4]  Combining  this  statement 
with  the  ideas  set  forth  in  defining  evaluation,  we  may  look 
at  evaluation  as  a  judgement  or  something,  say  a  program, 
with  the  purpose  of  improving  the  current  attainment  of  that 
program's  goals  or  otjectives.  This  position,  though,  seems 
to  make  evaluation  a  method  of  program  improvement  rather 
than    a   tool     to    help    achieve   this   end.  The    judgement   made 

may    indicate      some    action    which      should    be    taken      to    improve 
the    organization's    goal   attainment,    but   the    judgement   in   and 
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of  itself  dees  net  cause  the  organization's  goal  attainment 
to  improve.  As  such,  the  evaluation  is  a  -col  for  program 
improvement.  Evaluation  as  a  tool  for  decision  making  is 
brought  cut  by  Anderson  and  Ball.  Their  use  of  the  phrase 
"...to  contribute  tc  decisions..."  [Ref.  5]  in  describing 
evaluation  makes  clearer  the  idea  that  evaluation  is  a  tool 
rather   than   an    end    in   itself. 

If  the  above  purposes  of  evaluation  are  accepted, 
then  we  may  wish  tc  form  a  new  definition  of  evaluation. 
This  definition  takas  into  account  evaluation's  purpose. 
Aggregating  the  previously  cited  authors'  opinions  and  defi- 
nitions we  may  look  at  evaluation  as  a  judgement  of  seme 
program  with  the  purpose  of  contributing  to  decisions 
concerning  the  current  attainment  of  that  program's  goals  or 
objectives. 

B.       PRINCIPLES    OF    EVALUATION 

There  appears  tc  be  a  general  acknowledgement  among 
authors  of  evaluation  literature  that  a  group  of  principles 
exists  which  governs  the  conduct  of  evaluations.  Tracey 
[Ref.  6]  listed  six  principles  which  may  be  found  in  various 
forms  in  the  writings  of  other  authors  (Ref.  1,  4,  5,  8,  9]. 
Evaluation    must: 

1 .  Be  con  ducted  in  terms  of  purposes,  that  is  the 
object ives  must  be  known .  If  the  objectives  are  not 
known,  the  evaluation  effort  cannot  measure  how  well 
they  are  being  attained. 

2.  Be  cooperative .  Cooperation  of  all  organiza- 
tional levels  is  essential.  Without  free  communica- 
tion, evaluation  results  will  not  reach  all  parties, 
diluting   their  usefulness. 

3-  3e  continuous.  Evaluation  must  be  an  on-going 
process  to  accurately  track  performance  and  aid 
planning   in    light  of   current    oojective  attainment. 
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4.  Be  soeci  fie.  Generalizations  are  not  as  useful 
as  specific  information  in  providing  performance 
information. 

5-  Provide  means  and  focus  to  appraise  self.  trac- 
tive and  product.  The  evaluation  must  provide 
information  cf  sufficient  quantity  and  specificity 
tc  evaluate  not  only  the  program  output,  but  the 
mechanism  of  converting  inputs  to  output  and  the 
individuals'  performance  within  the  mechanism. 
6.  Be  based  en  uniform  and  objective  methods  and 
standards.  Methods  and  standards  which  change  from 
one  evaluation  to  the  next  destroy  trust  and  leave 
these  being  evaluated  questioning  how  they  should 
perform    their  work    tasks.      [Ref.    6:p.  14-15] 

C.       APPROACHES    TO    EVALOATIOH 

Hew  does  one  approach  or  categorize  evaluation?  The 
following  section  discusses  eight  approaches  to  or  catego- 
ries   cf   evaluation    forwarded   by   House   [Ref.    1:p. 21-43]. 

1  •      The   Systems    Analysi s    Approach 

The  systems  analysis  approach  defines  a  small  number 
cf  output  measures  and  attempts  to  relate  differences  in 
programs  tc  variations  observed  in  the  variables.  The  data 
acquired  through  this  observation  is  quantitative  in  nature. 
Correlational  analysis  or  other  statistical  methods  are  used 
to  relate  the  output  measures  to  the  programs  being  evalu- 
ated. This  method  is  widely  used  in  the  Department  of 
Health,  Education  and  Welfare  in  evaluating  federal  social 
welfare   programs. 

An  example  is  the  Office  of  Economic  Opportunity 
(0E0)  evaluation  of  the  Neighborhood  Health  Center  (NHC) 
program.         The    020      defined      five   areas      of      interest    to      be 
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investigated  in  determining  the  impact  of  the  NHC's.  These 
areas   of   interest   were: 

1.  Success   of    the   NHC's    in    providing    comprehen- 
sive  health   care   to   the    poor. 

2.  Patient    reaction   to   the    care   received   at   the 
NHC*  s. 

3.  Degree     of    implementation      of  comprehensive 
and    continuous    family  care   at   the   NHC's. 

4.  Functional    and  organizational  comparison  of 
the    NHC's. 

5.  Antipoverty      consequences    of      NHC    services. 
[Ref  .    7:p. 107-121  ] 

The  NHC  prcgram  was  evaluated  according  to  the  attainment  of 
the  objectives  which  relate  to  the  five  specified  interest 
areas . 

Cne  problem  which  may  be  seen  with  this  approach  is 
ensuring  the  output  measures  selected  truly  reflect  the 
organization's  goals.  If  the  selected  measures  do  net  accu- 
rately reflect  those  goals,  the  outcome  of  this  approach  may 
be    of    liiited  use. 

2 .      The    Eehavior al-0 bjectives    (Or    Goal-Based)     Approach 

This  approach,  popularized  in  business  and  govern- 
ment organizations  as  management  by  objectives,  uses  the 
stated  gcals  of  a  program  as  the  output  measure  and  evalu- 
ates prcgram  success  by  the  attainment  of  these  goals.  It 
can  be  seen  that  this  method  of  evaluation  addresses  only 
the  issue  of  program  effectiveness,  providing  no  information 
on  prcgram  efficiency.  In  this  sense,  effectiveness  is  a 
measure  of  the  extent  to  which  an  organization's  objectives 
are  achieved.  Efficiency  refers  to  the  cost  of  converting 
program  inputs  to  outputs,  that  is,  the  cost  of  objective 
achievement.  An  early  advocate  of  this  behavioral-objective 
approach  was  Tyler  [Ref.  8]  who  advanced  this  method  for 
evaluating    educational  goals   in   terms   of   student    behaviors. 

16 


Eeter  F.  Drucker  popularized  the  term  "management  by 
objectives"  in  his  beck  The  Practice  of  Management  [Ref.  9]. 
Implementation  of  management  by  objectives  (M30)  forces 
individuals  and  organizations  to  define  specific  areas  of 
responsibility  in  terms  of  measureable  expected  results, 
called  objectives.  Performance  is  determined  by  comparing 
objective   attainment     against   the      objectives   stated.  The 

popularity  of  the  approach  can  be  seen  in  its  widespread 
use.  A  1S76  study  showed  41  percent  of  the  hospitals 
surveyed  us^d  M EO  and  another  33  percent  were  planning  to 
start  in  the  near  future  [Ref.  10:p.8-11].  HBO  is  used  not 
only  as  an  evaluation  approach,  but  as  a  means  of  planning, 
coordination,  communication  and  control.  An  advantage  is  the 
explicit  statement  of  objectives  which  let  workers  know 
their  specific  duties  and  encourages  communication  between 
workers      and  supervisors      relating  to      job    performance.  A 

major  disadvantage  is  the  problem  of  specifying  behaviors 
rather  than  performance.  Specific  objectives  are  very 
measureafcle,  but  behaviors  are  not  necessarily  measureable 
in  the  context  of  contributing  to  goal  attainment.  Waks 
[Ref.  1:p.U87]  argued  that  "...acting  with  purpose..."  is 
not  eguivaient  to  "...taking  means  to  a  well  defined  end." 
In  other  words,  though  a  specified  behavior  may  be  observed, 
it  does  net  follow  that  this  behavior  leads  to  a  desired 
objective. 

3 .      The    De c i si o n - Mak in q    Approach 

As  an  earlier  definition  of  evaluation  implied, 
evaluation      is      closely   related      to      decision-making.  The 

decision-making  approach  holds  that  an  evaluation  is  struc- 
tured according  to  the  decisions  which  must  be  made.  It 
assumes  that  the  decision-maker's  concerns  are  the  signifi- 
cant areas  the  evaluation  must  address.  3y  structuring  the 
evaluation    in  this    manner,      the  results    should   be    of   greater 
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use  to  the  decision- maker.  This  approach  relies  heavily  on 
survey    methods    such    as  interviews   and   questionnaires. 

Stufflsbeam  et  al.  [Ref.  **],  whose  previously  cited 
definition  of  evaluation  includes  the  idea  that  evaluation 
is  to  provide  information  for  judging  decision  alternatives, 
is  an  advocate  of  this  approach  in  the  field  of  education. 
The  evaluation  is  structured  with  respect  to  the  decision- 
makers1 concerns  and  position  in  the  organization,  and 
specific  evaluation  subtasks  are  identified  and  assigned. 
The  results  of  these  subtasks  are  aggregated  and  communi- 
cated to  the  decision-maker  in  order  to  aid  in  the  decision 
process.  [Ref.  4]  This  approach  relieves  the  evaiuatcr  from 
having  tc  guess  the  audience  of  the  evaluation,  thereby 
providing  structure  for  the  entire  evaluation  effort.  On  the 
ether  hand,  this  approach  assumes  that  the  decision  maker's 
goals  are  the  same  as  those  of  the  entire  organization, 
which   may   cr  may   not    be   the    case. 

4.      The    Goal -Free   Approach 

Each  of  the  previously  discussad  approaches  involved 
program  evaluation  in  terms  of  program  goals  and  specific 
goals  for  the  evaluation.  The  goal-free  approach  seeks  to 
conduct  evaluation  in  terms  of  program  goals  without  refer- 
ence to  the  goals  for  the  evaluation,  indeed,  the  evaluator 
is  purposely  kept  unaware  of  these  goals  so  as  net  to  be 
tiassd    by   them. 

Scriven  [Ref.  1 1 ],  a  leading  proponent  of  this 
school  of  thought,  feels  that  the  goal-free  approach  is  a 
valid  method  of  reducing  bias  in  evaluation,  since  knowledge 
of      evaluation      goals     can    influence      the      evaluator.  For 

example,  an  evaiuatcr  who  is  tasked  with  conducting  a 
performance  evaluation  of  an  employee  with  the  explicit 
intent  of  determining  whether  the  employee  should  be  termi- 
nated  may      deliver   a    different      evaluation    if   the      intent   is 
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not  stated.  In  the  fcrmer  instance,  evaiuator  knowledge  that 
his  evaluation  may  result  in  a  worker  losing  his  jcb  may 
bias  the  outcome  of  the  evaluation.  3y  being  unaware  of  the 
evaluation  intent,  the  latter  situation  may  result  in  a  more 
accurate    representation   of    the    worker's   performance. 

This  approach  is  widely  used  in  the  area  of  consumer 
product  evaluations.  Various  consumer  organizations  regu- 
larly evaluate  products  placed  in  the  market  without  know- 
ledge of  the  manufacturers  goals.  These  evaluations  stress 
standards  and  criteria  which  they  (the  consumer  organiza- 
tion) feel  are  beneficial  to  the  consumer.  One  main  problem 
to  overcome  in  this  approach  is  the  choice  of  evaluators. 
Scriven  [Bef.  11]  sees  evaluators  as  experts,  able  to  elimi- 
nate and  prevent  both  self-bias  and  bias  of  others  from 
impacting  on  the  evaluation.  A  variety  of  techniques,  such 
as  cedes  of  ethics  or  double-blind  experiments,  are 
available   to  assist    the   evaluator    in    eliminating    bias. 

5  *      The    Art    Criticism  Approach 

This  approach  relies  upon  the  critic  to  make  judge- 
ment en  a  program  much  the  same  way  an  art  critic  would 
judge  a  fine  painting.  Though  opinions  on  specific  details 
may  vary,  there  is  generally  a  consensus  among  critics  of  a 
certain  endeavor  as  to  what  constitutes  a  notable  woric.  This 
implies  an  extensive  base  of  common  knowledge  among  those 
eligible   to   conduct    such   criticism. 

Eisner  makes  a  distinction  between  connoisseurship 
and  criticism.  While  connoisseurship  is  "recognizing  and 
appreciating  the  qualities  of  the  particular"  it  requires  no 
public  disclosure  or  judgement.  Criticism  necessarily  encom- 
passes connoisseurship.  "Criticism  is  the  art  of  disclosing 
the  qualities  of  events  or  objects  that  connoisseurship 
perceives."     [  Ref.     12   :p.197] 

The  key  purpose  of  criticism  is  to  increase  aware- 
ness   of    a   subject   area  and      convey      judgements      in      terms  of 
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criteria  which  are  accepted  among  -hose  knowledgeable  in 
that  area.  It  allows  the  uninitiated  to  gain  an  appreciation 
for  that  area  through  the  critic's  knowledge.  Though  gener- 
ally associated  with  art,  literature  and  other  basically 
creative  areasr  the  art  criticism  approach  to  evaluation  has 
teen    applied  to    the    field  of   education   with   some   success. 

A  key  problem  with  this  approach  is  generating 
acceptance  cf  the  critic's  criteria  for  judging  a  program.  A 
critic  nay  possess  extensive  knowledge  in  his  field,  but  if 
the  audience  of  his  evaluaton  is  not  receptive,  his  criti- 
cism   is    ret    likely    tc  carry    much    weight. 

6 .  The    Professional   Review    (Accreditation)    Approach 

The  professional  review  approach  has  some  distinct 
parallels  with  the  art-criticism  aproach  immediately  above. 
Professional  review  relies  upon  expert  opinion  concerning 
generally  accepted  standards  of  performance  in  evaluating  a 
particular  area.  The  standards  here,  though,  are  usually 
more  easily  quantified,  leading  to  a  more  structured 
approach  in  the  evaluation.  Professional  review  also  is  apt 
to  use  many  members,  organized  as  an  accreditation  or  review 
board  tc  conduct  the  evaluation.  Standards  and  measurement 
criteria  ars  determined  by  the  professionals  themselves  as 
they  are  accepted  as  the  experts  in  their  fields.  This 
approach  produces  an  evaluation  of  professionals  by  profes- 
sionals and  its  outccmes  are  not  easily  influenced  by  the 
layman. 

7 .  The   Qua si^Le^al    (Ad  versary)     Approach 

Cne  of  the  long  standing  approaches  for  evaluating 
and  policy-making  is  the  quasi-legal  approach.  It  is  an 
apprcach  tc  evaluation  which  closely  imitates  legal 
procedures.  Information,  or  'evidence',  concerning  a  program 
is   obtained    from    'witnesses',      much    as    testimony    is    received 
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in  a  court  of  law.  Information  both  for  and  against  a 
particular  program  is  presented,  and  great  care  is  exercised 
to  ensure  that  all  pertinent  information  is  received  after 
which  a  panel  of  evaluators  weighs  the  svidence  heard  and 
can  reach  a  decision  as  to  the  worth  of  the  program. 
Examples  of  this  approach  abound  in  today's  government, 
ranging  frcm  local  school  board  decisions  on  grade  schcol 
curricula  through  presidentially  appointed  panels  like  the 
Warren  Commission  which  investigated  the  assassination  of 
President    Kennedy. 

This  approach  does  not  rely  only  on  expert  evalua- 
tors as  have  several  previous  approaches.  Additionally  it 
not  cnly  accepts  but  encourages  personal  Dias  and  opinion  in 
those    providing    information.    As   Self    notes: 


The  ultimate  evidence  which  guides  deliberation  and 
judgement  includes  not  only  the  'facts1,  but  a  wide 
variety  of  perceptions,  opinions,  biases,  and  specula- 
tions, all  within  a  context  of  values  and  beliefs. 
[Ref.    13:p.21] 


The  ultimate  goal  of  this  approach  is  to  reach  a  definite 
conclusion  en  some  issue.  Its  conclusions  will  address  abso- 
lutes, such  as  'Is  the  program  meeting  its  goals'  rather 
than  matters  of  degree,  as  'To  what  extent  are  our  goals 
met '  . 

8.      The   Case   Study,    (or    Transaction)     Approach 

This  approach  is  widely  used  and  accepted  in  organi- 
zational studies.  It  focuses  on  program  processes  and 
interactions,  both  within  and  outside  the  program,  with  the 
intent  of  giving  the  reader  of  the  case  study  a  greater 
appreciation  of  the  program's  workings.  This  approach 
commonly  presents  interviews  with  people  in  the  program  and 
observations  made  by  the  interviewer  at  the  program  site  in 
the    form    cf    a  case.       The      case   can   be    examined   by    evaluators 
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and  conclusions  reached  through  discussions  and  sharing  of 
ideas  among  the  evaluators.  The  case  study  and  its  conclu- 
sions are  aimed  at  the  reader  who  does  not  possess  a  great 
knowledge  of  the  evaluation  area  as  a  means  of  increasing 
his/her  understanding  by  illustrating  how  others  view  the 
program  being  evaluated.  This  approach  allows  the  reader  to 
mere  fully  understand  the  internal  workings  of  the  program 
and   hew    program    inputs  are    converted    to   outputs. 

A  major  problem  with  this  approach  can  be  ensuring 
confidentiality  for  the  members  upon  which  the  case  study 
was      based.  Case      study      authors      may        have      difficulty 

disguising  all  of  the  personalities  involved  in  a  case. 
Another  problem  which  may  be  encountered  is  representing 
fairly  the  great  diversity  of  actions  and  opinions  which  a 
large  case  study  may  entail.  A  complicated  case  with  many 
personal  interactions  can  require  a  tremendous  editorial 
effort   to   ensure  that   it    is    accurate    and   understandable. 

9 .      Summary 

The  above  approaches  are  certainly  not  all  inclu- 
sive, nor  can  all  approaches  to  evaluation  be  expected  to 
fit  into  these  eight  categories.  They  are  intended  to  show 
the  variety  cf  approaches  available  in  conducting 
evaluations.  Though  the  overall  purpose  of  evaluation  may  be 
the  same,  that  is  providing  information  to  aid  in  decision 
making,  different  situations  may  call  for  different 
approaches  to  provide  necessary  information.  The  eight 
approaches  show  that  techniques  can  be  chosen  to  fit 
evaluation  to  evaluatcr  skill  (quasi-legal  vs.  professional 
review  approaches)  ,  program  objectives  (system  analysis  vs. 
behavioral-objectives     approaches),  or      even        to      ignore 

evaluation   objectives    (goal-free    approach)  . 
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D.       HHEN    TO    EVALUATE 

Stufflebeam  et  al.  [Ref.  4]  provide  a  view  of  evalua- 
tion which  investigates  when  in  the  program  life  cycle  eval- 
uation is  tc  take  place.  They  have  defined  four  types  of 
evaluaticn--context ,  input,  process,  and        product 

evaluaticn--which  serve  functions  from  program  inception 
through  final  impact  on  the  system  in  which  the  program 
operates.    Each    evaluation   type   is   explained   briefly    below. 

1  •      Context    Evaluation 

Context  eva  lust  ion  is  used  in  the  plan nine  orcce^s 
with  the  iii-Lwi. t  of  Identifying  ai..«et  .oils  o.  a. .used  opp<j_  - 
tunities  and  identifying  problem-  which  prevent  the  goals 
~ r c  iii  ce rr g  in et  ^r  t  *.  e  o ppo r  i  ir.i t ie s  tr  o m  l  e i.r.  j  q  s e q •  i  ni  ~ 
problem  identification  leads  tc  formulation  of  program 
objectives  which  are  used  as  yardsticks  against  which 
program  performance  is  measured.  Stufflebeam  et  al. 
[Ref.  4]  further  identify  two  modes  of  context  evaluation: 
contingency  and  congruence.  The  contingency  mode  locks 
cutsid=  the  system  for  factors  which  may  yield  improvements 
withir..  Typically,  if-then  type  questions  relating 'outside 
factors  tc  objectives  are  asked — if  our  manning  level  is 
reduced  by  20 %  then  can  we  carry  out  our  mission?  If 
research  costs  continue  to  rise,  then  is  our  present  budget 
adequate?  Congruence  mode  is  a  comparison  between  goals  and 
actual  performance.  This  mode  informs  the  organization  as  tc 
its  gcai  attainment.  As  opposed  to  contingency  mode,  congru- 
ence mode  looks  only  within  the  system  in  question  to 
provide    evaluation    data. 

2 •      Ingut   Evaluation 

Input  evaluation  is  concerned  with  the  use  of  avail- 
able resources  in  obtaining  objectives  formulated  in 
context    evaluation.     It   is   useful    in    providing    information   to 


rtruotur  i^y  ..ie  proyrd.k,  s.::a  its 
compared  to  a  ccst/benefit  analysis  with  resource  usage  =- 
the  cost  and  goal  attainment  as  the  benefit.  Besides  program 
structuring,  input  evaluation  also  helps  address  such  prob- 
lems as  the  need  for  additional  resources  and  other  general 
strategic   decisions. 

3  •      Frocess    Evaluation 

Process  evaluation  begins  after  program  approval  and 
implementation.  Process    evaluation      analyzes   the      program 

process  as  it  is  operating  to  provide  information  on  whether 
the    process     is    working     as    designed.  Stufflebeam    et      al. 

£Ref.  4]  point  out  that  this  type  of  evaluation  is  particu- 
larly important  early  in  program  implementation,  when  firm 
output  information  is  not  yet  available.  It  alicws  the 
organization  to  measure  hew  well  it  is  carrying  cut  the 
program    plan. 

^  •      Product    Evaluation 

Product  evaluation  provides  information  on  gcal 
attainment,  how  well  the  stated  objectives  are  met.  It  is  a 
major  input  to  decisions  which  would  modify  the  prcgram 
after   implementation. 

Ihe  view  provided  by  Stufflebeam  et  al.  [Ref.  4] 
should  net  be  regarded  as  an  evaluation  approach  different 
from  those  listed  by  House  [Ref.  1],  but  as  an  expansion  of 
those  approaches.  Each  of  the  eight  approaches  cculd  be 
structured  to  lock  specifically  at  input,  context,  process 
or  output  though,  as  implied  earlier,  the  different 
approaches  may  not  be  equally  effective  in  providing  infor- 
mation in  these  four  areas.  The  Stufflebeam  et  al.  view  can 
be  seen  as  helping  determine  the  timing  of  evaluations, 
using  one  of  House's  approaches,  to  provide  information  on 
specific    portions  of    a  program's   life-cycle. 
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E.       SOflMABY 

This  chapter  has  focused  on  the  many  ideas  and 
approaches  available  in  evaluation  science.  Definitions  of 
evaluation  and  its  purposes  were  presented  to  show  the  simi- 
larities and  differences  that  exist  among  authors  of  evalua- 
tion literature  and  a  definition  of  evaluation  was  formed. 
The  definition  looked  upon  evaluation  as  a  judgement  of  seme 
program  with  the  purpose  of  contributing  to  decisions 
concerning  the  current  attainment  of  that  program's  goals  or 
objectives.  Six  principles  for  evaluation  were  also 
presented,  demonstrating  hew  and  when  evaluation  should  be 
conducted  and  what  kind  of  information  should  be  prcvidsd  by 
the    evaluation. 

The  fcasic  concepts  of  evaluation  were  expanded  by  inves- 
tigating eight  approaches  which  are  available  to  ©valuators. 
These  approaches  provide  different  evaluation  structure 
depending  on  the  type  of  information  desired  from  the  evalu- 
ation or  the  different  evaluation  assets  availaoie. 
Finally,  a  view  of  evaluation  which  addresses  when  to 
perform  evaluation  was  added  to  the  eight  evaluation 
approaches. 

With  this  grounding  in  the  fundamental  ideas  of  evalua- 
tion, the  next  chapter  will  focus  on  the  evaluator's  roles 
and  responsibilities,  and  some  problems  associated  with 
evaluation.  The  evaluator's  implementation  of  the  above 
principles  and  methods  can  greatly  influence  the  eventual 
cutccme    cf    the    evaluation. 
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III.     E7AI-0AT0RS 


The  ideal  rater  who  observes  and  evaluates  what  is 
important  and  reports  his  judaement  without  bias  or 
appreciable  error"  dees  not  exist,  or  if  he  does,  we 
don't  know  how  to  separate  him  from  his  less  effective 
colleagues.      [  Ref  .     14:p.7] 


Though  the  above  statement  may  be  true,  many  steps  have 
been  taken  in  evaluation  science  to  identify  competent  eval- 
uators  ana  improve  performance  of  avaluators  in  general. 
This  chapter  locks  at  the  evaluator,  beginning  with  a 
discussion  of  objectivity  and  validity  as  they  relate  to 
evaluation,  who  performs  evaluations  and  whether  they  ccxe 
from  within  or  outside  the  organization  is  investigated, 
with  advantages  and  disadvantages  presented  for  each  evalua- 
tion source.  A  discussion  of  the  kinds  of  errors  ©valuators 
typically  make  is  presented  along  with  sources  which  may 
cause  these  errors.  The  chapter  closes  with  a  discussion  of 
several  methods  for  reducing  the  amount  of  errors  evaluators 
may  bring  into  their  evaluations,  ranging  from  training  the 
evaluator  tc  improving  the  tools  the  evaluator  uses  in 
performing    evaluation. 

A.       OBJECTIVITY 

Objectivity,  in  the  context  of  evaluation,  is  the 
ability  tc  observe  something  only  as  it  physically  exists 
without  the  inclusion  of  personal  feelings  about  the  object. 
For  example,  the  statement  'Joe  is  six  feet  tall1  would  be 
considered  mere  objective  than  saying  'Joe  is  a  giant'.  The 
former  could  be  adequately  demonstrated  using  a  tape 
measure,  while  the  latter  is  largely  dependent  upon  the 
particular  observer's  concept  of  what  is  giant  and  what  is 
not.    As    House   points    cut: 
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;pecified    procedures  for    obsarvation.    3y 

objectivity      is   achieved      by    having      observers    agree      on 
what  they  see — replication  of  observation. 

[Ref.    1:p.215] 


House  calls  this  the  quantitative  notion  of  objectivity. 
The  concept  of  reliability  in  observation  closely  parallels 
this  quantitative  nction.  Reliability  is  based  on  the 
ability  tc  replicate  observations.  That  is,  if  a  particular 
observation  of  an  object  can  be  replicated,  that  observation 
is    assumed   to   be   reliable. 

B.       VALIDITY 

The  concept  of  validity  is  important  to  evaluation.  If 
an  observation  does  net  accurately  reflect  the  qualities  of 
an  object  one  wishes  to  measure,  a  'true'  evaluation  of  that 
object    may    be   impossible.  Scriven    [Ref.    15]  addresses   the 

concept  of  validity  by  bringing  cut  a  feature  which  he  calls 
the  qualitative  sense  of  objectivity.  He  argues  that,  taken 
in  the  extreme,  the  quantitative  notion  of  objectivity 
confuses  the  method  of  verification  with  'truth*.  An  obser- 
vation may  te  widely  agreed  upon  and  r eplicat eable ,  but  how 
closely  dees  it  represent  reality?  How  'good'  is  the  obser- 
vation? To  illustrate,  Scriven  cited  the  incident  of  a 
television  receiver  evaluatcr  observing  picture  quality.  The 
evaluatcr  used  a  mechanical  device  to  measure  decibel  gain 
of  the  receivers,  though  there  was  little  correlation 
between  decibel  gain  and  picture  quality.  The  observations 
obtained  were  able  tc  be  replicated  and  the  results  widely 
agreed  upon  but  they  did  not  really  relate  tc  picture 
quality.  In  this  case,  the  evaluation  was  quantitatively 
reliable      but      lacked     quality.  [Ref.    15]      The      issue     of 

evaluation   quality    is  commonly    referred    to    as   validity. 
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As  a  method  of  relating  observations  to  objects  w=  wish 
to  evaluate,  Cummings  and  Schwab  [Ref.  16]  suggest  the 
concept  cf  construct  validity.  A  construct  is  a  mental  image 
we  have  of  something,  the  way  we  perceive  something. 
Validity,  in  this  context,  refers  to  the  correlation  between 
our  mental  image  and  some  measure  of  it.  In  the  previous 
example,  there  was  little  correlation  between  decibel  gain 
of  the  television  receivers  and  quality  cf  the  picture  hence 
there  was  little  construct  validity.  A  different  measure 
which  mere  closely  corresponds  to  our  mental  image  of 
picture  quality  could  be  chosen.  The  closer  the  measure 
chosen  corresponds  with  our  mental  image  of  something,  the 
greater  the  construct  validity.  A  different  measure  such  as 
viewer  satisfaction  will  have  varying  degrees  of  construct 
validity  according  tc  hew  closely  it  compares  with  our 
mental    image  of    picture    quality. 

Tc  better  illustrate  the  concept  of  construct  validity, 
consider  Figure  3.1.  As  shown,  the  ieft  circls  represents 
seme  construct  we  are  interested  in  and  the  right  circle 
represents  some  measure  of  that  construct.  Ideally,  there 
would  be  ccaoplete  cverlap  cf  the  circles  representing  a 
total  correlation  between  the  construct  and  the  measure 
used.  There  are  two  general  reasons  that  the  two  circles  do 
not        completely  cverlap  —  measurement        deficiency  and 

measurement    contamination  (Ref.    16]. 

Measurement  deficiency  occurs  when  the  measure  fails  to 
take  into  account  all  of  the  factors  present  in  our 
construct.  For  example,  a  measure  of  a  data  processing 
departments  performance  which  accounted  for  quantity  of 
output  but  neglected  quality  and  timeliness  would  probably 
be   considered  deficient. 

Measurement  contamination,  in  contrast  to  measurement 
deficiency,  occurs  when  the  measure  takes  into  account 
factors    which   fall    outside    cur    construct.      If   our    measure   of 
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Adapted    from   [rsf.17    :p.75] 


Figure    3, 1        Deficiency   and   Contamination. 

the  data  processing  department's  performance  includes  items 
such  as  corporate  sales  or  top  management • s  perceptions  of 
the    department,    the    measure    is    likely    to   be   contaminated. 

It  may  be  seen  that  both  deficiency  and  contamination  in 
measurement  of  constructs  adversely  affect  construct 
validity.  If  cir  measures  do  not  contain  ail  the  factors 
pertinent  to  cur  construct,  or  if  the  measures  contain 
factors  outside  our  construct,  it  is  unlikely  that  the 
measures  will  accurately  reflect  the  mental  image  of  the 
construct.  Both  of  these  circumstances,  then,  decrease 
construct   validity. 

C.       EPFOES 

There  are  a  number  of  errors  which  evaluatcrs  may  commit 
during  the  evaluation  process.  Cummings  and  Schwab  [Ref.  16] 
discuss  these  errors  in  two  main  groups-  variable  error  and 
constant  error.  These  two  groups  are  explained  below,  with 
examples. 
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1  •      Variable   Srrcr 

Variable  srrcr  is  e valuator  disagreement  which  mani- 
fests itself  as  differences  in  the  scores  of  specific  items 
cf  an  evaluation.  It  may  take  two  forms  —  disagreements 
between    evaluators    and  disagreements    over   time. 

a.      Disagreements   between   evaluators 

Suppose  two  evaluators,  A  and  B,  have  cbserved 
five  workers  performing  their  jobs  and  rated  the  workers' 
performance  on  a  scale  of  0  (pcor  performance)  to  10  (high 
performance).  The  ratings  are  shown  in  Table  I.  Note  that 
there  is  total  rating  agreement  only  on  worker  4  and  the 
other   ratings   differ    from    1    to    4    units. 

TA3LE    I 
Evaluator   Ratings 
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RATINGS 
^VALUATOR  A      EVALUATOR  3 


3 
8 
7 
9 
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Taking  the  ratings  obtained  from  A  and  E,  we  now 
wish  to  plot  the  scores,  with  evaluator  A's  rating  repre- 
senting the  X -component  of  our  plot  and  evaluator  3's 
ratings  representing  the  Y-ccmponent  of  the  plot.  The 
result  is  a  graph  as  shown  in  Figure  3.2.  The  straight  line 
extending  from  the  origin  and  rising  from  left  to  right 
represents  total  agreement  between  the  evaluators.  The 
distance  of  each  worker's  score  from  the  total  agreement 
line  is  a  measure  of  the  disagreement  between  the 
evaluators.  A  linear  correlation  coefficient  may  be 
calculated    which   expresses    the   amount    of      agreement      between 
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the  evaluatcrs.  Values  for  the  linear  correlator  coefficient 
may  vary  from  -1.0  (highly  negative  correlation,  meaning 
that  high  values  for  the  X-component  -end  to  go  with  low 
values  for  the  Y-ccmponent  and  low  values  for  the 
X-component  tend  to  gc  with  high  values  for  the  Y-component) 
to  +1.0  (highly  positive  correlation,  meaning  that  high 
values  fcr  the  X-component  tend  to  go  with  high  values  for 
the  Y-ccmponent  and  low  values  for  the  X-component  tend  to 
go  with  lew  values  for  the  Y-component)  ,  with  a  value  of  0.0 
indicating  nc  correlation  (no  predictable  pattern).  In  this 
example,  the  linear  correlation  coefficient  is  0.6  indi- 
cating seme  positive  correlation  between  evaluatcrs  A  and  B. 
A  value  ir.  the  range  cf  0.8  to  0.9  would  tend  to  indicate  a 
strong  .rcrrc.iaticn  between  A  and  3.  High  correlation  does 
not,  however,  guarantee  a  valid  rating.  It  simply  shows  that 
A  and  3  agree  on  what  they  have  observed.  3oth  A  and  B  may 
be  wicng  in  their  ratings  of  worker  4,  but  their  agreement 
would  provide  some  confidence  that  their  rating  was  correct. 
Two  methods  which  can  reduce  disagreement 
tetween  evaluators  are  redaction  or  elimination  cf  subjec- 
tivity in  measurement  instruments  and  ensuring  evaluator 
familiarity  with  the  job  being  evaluated.  The  former  method 
reduces  disagreements  by  relieving  the  evaluator  cf  inter- 
preting subjective  measures.  By  using  more  objective  evalua- 
tion measures,  evaluator  bias  is  less  likely  to  be 
accidentally  introduced  [Ref.  20  :p.46].  Ensuring  evaluator 
familiarity  with  the  job  being  evaluated  increases  the  like- 
lihood of  evaluating  jcb  factors  which  correlate  highly  with 
job    performance. 

fc.      Disagreements  Over   Time 

Disagreements  over  time  pertain  to  disagreements 
in  evaluations  made  by  one  evaluator  at  different  points  in 
time.      Suppose    that,    in   the    example    of    disagreements   between 
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E  VALUATOR    A    RATINGS 
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Figure   3.2        Evaluator   Disagreements. 

evaluators,  evaluatcr  A's  ratings  represented  an  evaluation 
performed  by  A  at  time  1  and  that  evaluator  B's  ratings 
represented  an  evaluation  performed  by  A  at  time  2. 
Calculation  of  the  linear  correlation  coefficient  would  then 
measure    hew    well  evaluator    A's   ratings   agree   over    time. 

Using  disagreements  over  time  as  a  measure  of 
construct  validity  is  generally  not  as  desireable  as  using 
disagreements  between  evaluators.  The  reason  for  this  is 
that  differences  in  evaluations  mada  at  different  points  in 
time  may  be  due  to  performance  improvament  or  degradation  of 
those      being     evaluated.      The      low      correlation      coefficient 


32 


obtained  from  a  comparison  of  evaluations  made  on  a  worker 
whose  performance  has  changed  markedly  ever  time  may  be 
mistakenly  taken  to  mean  the  construct  is  not  valid*  For 
this  reason,  correlation  coefficients  obtained  by  comparing 
two  cr  more  evaluatcrs'  ratings  are  a  better  measure  of 
construct  validity  [Ref.  16].  A  method  of  reducing  disa- 
greements over  time,  discussed  later,  is  testing  potential 
evaluatcrs  and  choosing  those  who  demonstrate  little  of  this 
error. 

2 •      £ c n s ta n t   Errors 

Where      variatle   errors      tend      to   create      differences 
between   evaluations,      constant    errors   tend   to   cause    spurious 
c"*mi]La~'''u";'cs»      Cc'sta."'"^    5 T*  r  o  r    *' k  as    *~  \~~  ^  ^    ^  c  ~  "n  s  —  —  bale    ©  r  ^ r"»~ 
central    tendency   and   leniency. 

a.  Hale  error 

Halo  error  occurs  when  an  evaiuator  fails  to 
differentiate  among  individual  items  or  dimensions  in  his 
evaluation,  but  evaluates  on  the  basis  of  his  overall 
impression.  The  boss  who  observes  only  an  employee's  written 
work  but  rates  the  employee  high  in  areas  such  as  initiative 
and    personal  relations  has    made   a   halo   error. 

b.  Central    tendency 

Central  tendency  is  the  tendency  for  evaluatcrs 
to  rate  all  dimensions  of  an  object  near  the  middle  of  the 
evaluation    scale,    avoiding    the    extremes. 

c.  Leniency 

This  errcr  is  committed  when  an  evaiuator  tends 
to  rate  all  objects  too  high.  The  'easy  grader'  consistently 
delivers  inflated  rating  marks.  The  opposite  error,  that  of 
rating   all   objects   tec  low    is   called    strictness. 
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Evaluator  training  in  the  area  of  constant  error 
is  a  useful  technique  in  reducing  these  errors.  A  discussion 
cf   this    technique   is    presented   in   a   later    section. 

E.       EVALUATION    SOURCES 

Evaluatcrs  may  ccme  from  many  places  within  and  outside 
an  organization.  Though  evaluations  by  superiors  are  very 
common,  alternative  sources  of  evaluation  exist--peer, 
subordinate,        self      and      disinterested        party      or      outside 

evaluatcrs. 

1  •      S u p e ri o r   Eva luators 

Evaluations  ty  superiors  are  a  widely  used  method  in 
xcday's  organizations.  Superiors  are  chosen  for  many 
reasons,  such  as  jot  experience,  familiarity  with  subordi- 
nate positions  and  jot  skills,  even  tradition.  Superiors  are 
often  the  legical  chcice  as  evaluatcrs,  for  their  position 
in  the  organizational  hierarchy  is  such  that  they  determine 
to  a  great  extent  the  incentive  and  reward  system  for  their 
subordinates.  As  such,  their  evaluations  cf  subordinates 
may  lead  to  direct  reward  cr  punishment  without  passing 
through  another  level  of  hierarchy  and  this  immediate 
evaluat icn-incentive  tie  keeps  subordinates  appraised  of 
their   performance. 

Seme  problems  can  exist  with  supervisor  evaluations. 
First,  if  the  subordinate  being  rated  does  not  work  directly 
for  the  evaluating  superior  or  if  there  is  suDstantial  phys- 
ical separation  of  the  supervisor  from  the  subordinate, 
supervisor  ebservatien  of  the  subordinate's  job  performance 
may  be  limited.  Alsc,  due  to  rapidly  changing  technology, 
the  superior  may  not  have  enough  understanding  of  the  subor- 
dinate's actual  on-the-job  responsibilities  to  adequately 
rate    his    performance.    Increasing   automation   in   the    workplace 
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tends  to  widen  the  "understanding  gap'  for  the  superior  who 
does  net  strive  to  stay  current  in  today^s  dynamic  business 
world . 

2  .      Peer  Eva  luat  crs 

Peer  evaluatcrs  are  those  individuals  who  work  at 
the  same  organizational  level  as  the  person  rated.  Many 
organizations  avoid  using  peer  evaluations,  dismissing  the 
technigue   as     a         •popularity      contest1.  Peer      evaluatcr- 

evaluatae  friendship  is  seen  as  biasing  the  validity  of  this 
technique.  This  may  be  due  to  the  perception  that  friends 
tend  to  minimize  or  overlook  one  another's  shortcomings  and 
only  elevate  good  points,  or  mistake  pleasing  personal 
attributes  for  indicators  of  high  job  performance,  Recent 
studies  (e.g.  Klimcski  and  London  [Sef.  17],  and  Love 
[Ref.  18]  )  have  shewn  that  evaluation  validity  is  not 
significantly  affected  by  friendship  bias,  and  that  in  some 
circumstances,  peer  evaluation  appears  to  offer  areat  bene- 
fits   tc    an    evaluation   program. 

3  •      Cis inter  est s  d   Pa  rt  y    ^valuator s 

Disinterested  parties  can  possibly  be  obtained 
within  the  organization  or  outside.  They  may  come  from  any 
organizational  level  so  long  as  they  have  no  vested  interest 
in  the  outcome  of  their  evaluations.  Seme  organizations 
bring  in  outsiders  tc  perform  this  function,  feeling  that 
lack  of  personal  contacts  within  the  organization  will  allow 
a   mere   objective   evaluation. 

A  problem  which  may  occur  with  disinterested  party 
evaluatcrs  is  that,  aside  from  having  no  vested  interest  in 
the  evaluation  outcome,  they  may  also  have  limited  insight 
into  the  factors  which  indicate  good  job  performance.  As 
noted  in  supervisor  evaluation,  the  evaluator  who  does  not 
stay    current  on      the    the   technology   of   the      workplace    is   not 
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likely  tc  deliver  as  good  a  performance  evaluation  as  one 
who  is  acre  familiar  with  that  technology.  In  addition, 
outsiders  brought  in  to  perform  evaluations  may  net  fully 
grasp  factors  such  as  organizational  politics  and  interper- 
sonal relationships  which  can  greatly  influence  overall  job 
performance. 

E.       DISCISSION 

Each  evaluation  source  has  unique  characteristics,  as 
well  as  similarities  with  each  of  the  other  sources,  in 
providing  evaluation  information.  Though  introduction  of 
evaluate!  errors  is  fairly  comparable  for  superior  and  peer 
evaluations  [Ref.  19],  studies  have  shown  that  rating 
sources  differ  in  their  perceptions  of  performance 
[Ref-  17].  This  difference  in  perceptions  is  related  to 
dimensionality. 

Dimensionality  is  the  quality  of  an  evaluation  area 
possessing  different  elements  or  dimensions.  For  instance, 
if  ore  examined  the  broad  area  of  secretarial  job  perform- 
ance, many  individual  dimensions  could  be  identified,  such 
as  typing  speed,  typing  accuracy,  shorthand  ability,  organi- 
zation, ability  to  speak,  effectively  on  the  telephone  and 
many  ethers.  These  dimensions  comprise  the  evaluation  area 
called   secretarial    jet   performance. 

Net  all  evaluation  sources  use  the  same  set  of  dimen- 
sions in  conducting  evaluations.  As  an  example,  consider  an 
evaluation  of  worker  performance  performed  by  a  worker's 
superior  an i  a  peer.  The  superior,  being  very  goal  oriented, 
rates  the  worker's  clerical  performance  according  to  how 
many  pages  are  typed  per  hour  assuming,  perhaps  incorrectly, 
that  guantity  of  pages  typed  also  indicates  quality.  The 
peer,  who  must  coriect  any  errors  made  by  the  worker,  is 
concerned  with  quality  of  output.  Different  sources  exhibit 
different      perceptions     of      performance.      Neither      view      is 
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necessarily  wrong,  but  this  illustrates  the  differences  that 
may  exist  between  evaluation  sources.  Holzburg  [Ref.  19]  has 
found  a  ccnsistent  outcome  of  dimensional  analysis  of  supe- 
rior and  peer  evaluations  is  that  evaluation  sources  deter- 
mine the  primary  dimensionality  of  the  evaluations.  What 
this  means  to  the  evaluatee  is  that  performance  grades 
received  may  be  due  more  to  the  evaluation  source  than  the 
job    performance. 

The  following  sections  discuss  seme  of  the  error  sources 
which  may  cause  evaluatcrs  to  commit  errors  and  methods  of 
reducing  various  errors  to  provide  more  accurate 
evaluations . 

P.       EBRCR    SOURCES 

Many  factors  contribute  to  evaluator  error.  Though  often 
grouped  under  the  general  heading  of  bias,  specific  factors 
have  teen  investigated  by  a  variety  of  study  groups  as  a  way 
of  ensuring  objective  and  valid  evaluations.  This  section 
looics  at  several  of  the  factors  contributing  to  evaluator 
error,  and  the  next  section  discusses  some  metnods  suggested 
for    reducing  these   errors. 

1  •      Social    Inter  action 

Social  interaction,  or  friendship  bias,  is  often 
cited  as  a  reason  for  avoiding  peer  evaluations.  As  previ- 
ously noted,  this  bias  is  thought  by  many  organizations  to 
adversely  affect  peer  evaluations.  This  bias  is  also  seen 
in  superior  evaluations,  but  judging  from  the  number  of 
organizations  which  use  superior  evaluators  as  a  primary 
means  of  evaluation,  the  effects  may  not  be  considered  as 
severe.  This  is  not  to  say  that  superior  evaluation  biases 
are  actually  less  severe  than  those  biases  found  in  ether 
evaluation    sources.       The    biases   may   be    just   as   bad,      but   the 
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superior's    position    tends  to   land   a    degree    of   credibility   to 
his    or   her    judgements,   deserved   or  not. 

2  .      E  va  lua  t  or   Inexperience 

Evaluator  inexperience  and  lacs  of  Training  in  eval- 
uation procedures  tend  to  contribute  to  haic  and  leniency 
errors  [Hef.  20]-  Fcorly  defined  measures  force  the  inex- 
perienced evaluator  to  make  interpretations  which,  due  to 
limited  background,  may  not  accurately  reflect  performance. 
Closely  associated  with  this  idea  is  the  evaluator' s  effec- 
tiveness en  the  job.  Low  evaluator  effectiveness  correlates 
strongly    with   low  evaluation   accuracy. 

3-      5c Is  Conflict 

A  strong  factor  contributing  to  evaluator  error  is 
the  rele  conflict  experienced  by  many  ^valuators.  Dayal  has 
noted: 


The  manager  has  to  accept  the  responsibility  to  judae 
the  performance  of  other  ceoole.  Often  this  responsi- 
bility is  aesitantly  rakeh  because  he  feels  uncomfor- 
table   in    his    role    as   judge.      [Hef.    21:p.29] 


One  affect  of  this  evaluator  discomfort  is  that  evaluation 
results  tend  to  group  near  the  upper  end  of  the  rating  scale 
[Hef.  21].  A  possible  reason  for  this  effect  is  that  giving 
low  ratines  may  result  in  slower  promotion  or  even  firing  of 
an  employee,  for  which  the  evaluator  giving  the  ratings  may 
feel  responsible.  Eatings  at  the  high  end  of  the  scale 
reduce  the  probability  that  employees  will  experience  lay- 
offs or  slower  promotion  and  the  evaluator  will  feel  less 
responsible    if    such    actions    do   occur. 

**  •      Evaluator   Knowledge    of   Evaluation    Purpose 

As        previously     stated,  Scriven      [Ref.    11]        has 

suggested   that    evaluator   knowledge   of   the    evaluation   purpose 
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may  be  ancther  nonperformance  factor  influencing  the  actual 
performance  rating  received.  A  study  by  Gallagher  [Rsf.  22] 
investigated  whether  ratings  of  performance  varied  when 
evaluatcrs  were  given  different  purposes  for  the  evalua- 
tions. The  results  support  Scriven's  contention.  Gallagher's 
discussion  of  the  results  concludes  "...that  a  single 
performance  evaluation  should  not  be  used  for  different 
purposes  since  the  stated  purpose  of  the  evaluation  can 
affect   the   actual   performance   rating."      [Bef.    22:p.38] 

G.       EBECB    REDUCTION    TECHNIQUES 

Many  techniques  are  available  to  help  reduce  evaluator 
error.  These  techniques  have  been  investigated  by  various 
evaluation  researchers  (e.g.  Bernardin  [Hef.  23],  Wiley  and 
Jenkins  [Ref.  24],  and  Scott  [Hef.  20]  )  and  some  suggested 
solutions   are  presented    here. 

1  •      Jv§iM.i.2Z  Training 

Eernardin,  in  a  study  of  comprehensive  vs.  abbrevi- 
ated evaluator  training  programs  found  that  evaluators 
"...trained  en  error  prior  to  observation  and  who  used  the 
scales  to  maintain  observational  diaries  had  significantly 
less  leniency  error  and  halo  effect  than  all  other  groups." 
[Ref.  23:p.302]  In  this  study  comprehensive  training  was  a 
one  hour  session  consisting  of  definitions,  graphic  illus- 
trations and  examples  of  halo  error,  leniency  and  central 
tendency  was  presented  to  students  who  were  acting  as  evalu- 
ators of  peer  performance.  The  trainees  were  also  given  data 
to  evaluate  in  terms  of  the  errors,  and  the  evaluations  were 
discussed.  Abbreviated  training  was  a  five  minute  session 
with  definitions  of  the  error  types  and  a  single 
illustration  of    each. 
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The  results  of  this  study  indicated  that  the  psycho- 
metric guality  for  those  who  underwent  comprehensive 
training  was  superior  to  those  who  received  abbreviated 
training  at  the  first  rating  period,  and  both  training 
groups  were  superior  to  the  control  (untrained  group). 
Another  result  was  that  the  positive  effects  of  the  training 
programs  were  virtually  nonexistent  after  one  additional 
ratirg  pericd-  (Ref.  23]  One  might  argue  that  for  an  organ- 
ization contemplating  a  training  program  for  supervisory 
personnel  the  above  information  may  indicate  that  a  compre- 
hensive training  program  would  lead  to  fewer  evaluator 
errors  than  an  abbreviated  training  program.  As  the  effects 
of  both  training  programs  tends  to  rapidly  diminish  with 
time,  however,  a  shorter  training  program  regularly 
administered  may  deliver  mere  positive  effects  in  the  long 
run. 

2«      £jsJ8SS§.iP..3ski    Aral  vsi  s 

As  discussed  previously,  different  evaluation 
sources  perceive  performance  in  different  ways.  To  account 
for  this,  subjective  evaluation  areas  should  be  examined  by 
dimensional  analysis.  This  analysis  is  used  to  investigate 
the  many  dimensions  which  comprise  an  evaluation  area  and 
considers  the  different  combinations  of  dimensions  used  by 
various  evaluation  sources.  Since  each  evaluation  source 
tends  to  use  different  dimensions  in  performing  evaluations 
(Ref.  25:p.U73]#  dimensional  analysis  can  provide  insight 
into  the  particular  concerns  of  the  various  sources. 
Klimcski  and  London  (Ref.  17]  present  the  example  that 
supervisors  may  be  less  able  to  discriminate  between  items 
related  to  competence  from  those  related  to  effort,  whereas 
nurses  rating  themselves  and  peers  can  make  that 
distinction.  This  would  suggest  that  supervisors  are  more 
likely   to   consider   effort   as   an    indicator    of   competence  than 
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peers.  By  accounting  for  the  dimensions  used  by  various 
evaluation  sources  dimensional  analysis  can  allow  perform- 
ance measures  to  be  tailored  according  to  the  anticipated 
evaluation  source,  or  it  may  be  used  after  the  fact  to  help 
explain  ratings  received  in  particular  areas  in  light  of  the 
evaluation   source. 

3  •      Testing    Evaluators 

Wiley  and  Jenkins  [ Ref .  24]  had  109  Air  Force  navi- 
gator students  estimate  qualifications  needed  to  perform 
various  air  Force  tasks  using  an  experimentally  standardized 
task  list  and  sets  of  five  rating  scales.  Their  estimates 
were  aggregated  and  a  consensus  or  pooled  estimate  group  was 
formed.  These  students,  after  one  month,  again  estimated 
qualifications  and  the  students  were  scored  by  correlating 
their  estimates  with  the  key  of  pooled  estimates.  The  study 
shows  that  evaluators  who  tend  to  agree  with  the  consensus 
also  tend  to  retest  self-agreement.  These  evaluators  also 
tend  toward  consensus  agreement  on  later  evaluations. 
[Ref.    24] 

The  above  findings  tend  to  suggest  that  a  standard- 
ized test  could  be  developed  to  rate  potential  evaluators.  A 
consensus  key  which  corresponds  to  the  organization's  view 
cf  performance  wculd  make  it  possible  to  select  evaluators 
with        corresponding      views.  This        would      help        ensure 

organizational  goals  are  being  pursued  by  the  evaluation 
process. 

**  •      Reducing   Sub  iectivity   of;    Evaluation   Measures 

Performance  appraisal  systems  are  commonly  regarded 
as  being  too  subjective  in  nature,  relying  primarily  on 
human  judgement  for  gathering  information  pertaining  to 
measures   [Hef.    20].  Elimination  of   all    factors      which   can 

not    be   objectively    measured    would      naturally    lead    tc   minimal 
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subjectivity.  While  this  elimination  may  or  may  not  be 
possible,  it  is  possible  to  develop  a  system  where  the  eval- 
uator  reacts  to  stimuli  which  are  relatively  free  of  subjec- 
tive or  irrelevant  influences  rather  than  stimuli  which 
require  the  evaiuatcr*s  judgement  [  Ref  •  16:p. 89-92].  The 
stimuli  take  the  form  of  actual  on-the-job  incidents  which 
the  evaluatcr  simply  observes  without  interpretation.  These 
incidents,  Dr  'critical  behaviors1 ,  represent  actions 
normally  associated  with  outstandingly  successful  or 
outstandingly  unsuccessful  za.sk  performance.  The  evaluator 
in  this  role  acts  as  a  reporter  of  actions  rather  than  a 
judge   whc   values   actions  [Ref.    20]. 

One  problem  associated  with  this  method  is  the 
choice  cf  critical  incidents  or  behaviors.  Some  perscn  or 
group  of  pecple  must  be  designated  to  decide  what  incidents 
are  tc  be  used  in  evaluation.  Providing  a  list  of  such 
incidents  reduces  the  evaluator1 s  need  to  exercise  personal 
judgement    in  conducting   evaluations. 

H.        SOHMAHY 

This  chapter  has  investigated  the  evaluator  as  part  of 
the  scheme  cf  evaluation.  The  concepts  of  objectivity  and 
validity  were  introduced  and  explained  as  they  pertain  to 
evaluation.  Sources  cf  evaluator  error  were  then  discussed. 
Evaluatcr  errors  were  divided  into  variable  and  constant 
errors,  and  each  cf  these  areas  was  broken  into  specific 
error  types.  Various  evaluator  sources-  superior,  peer  and 
disinterested  party-  were  discussed  with  advantages  and 
disadvantages  of  each  source  considered.  A  discussion  of 
error  sources,  along  with  techniques  to  reduce  these  errors 
closes  the  chapter.  The  last  section  suggests  that  training 
and  testing  evaluatcrs  and  taking  measures  to  reduce  the 
subjectivity  of  evaluation  measures  can  have  a  significant 
effect  in  reduction  cf  evaluator  error. 
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The  next  chapter  uses  the  information  presented  in 
Chapters  II  and  III  to  analyze  the  MCCHES  and  offer  seme 
suggestions  for  identifying  and  controlling  or  controlling 
for  potential  evaluatoz  bias. 
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IV.    MCCRES 


The  curpcse  of  the  Marine  Corps  Combat  Readiness 
Evaluation  System  (MCCRES)  is  to  provide  a  timely  and 
accurate  evaluation  of  the  readiness  of  ?leet  Marine 
Forces,  including  Beserve  units,  to  accomplish  assigned 
missions.      [Ref.    26:p.I-A-1] 


To  achieve  the  objective  of  timely  and  accurate  readiness 
evaluation,  the  MCCFES  has  been  designed  to  allow  observa- 
tion of  Marine  units  in  simulated  combat  situations.  It 
promotes  use  of  a  standardized  evaluation  process  and 
reporting  system  to  provide  feedback  to  the  evaluated  unit 
indicating  strengths  and  weaknesses  in  a  combat  readiness 
posture.  This  chapter  focuses  on  the  evaluation  process  in 
an  attempt  to  identify  areas  where  avaluators  may  commit 
errors  or  inject  bias  into  the  evaluation  possibly  leading 
to  inaccurate  readiness  ratings.  The  general  evaluation 
approach  and  structure  of  the  MCCRES  are  discussed  first, 
followed  by  an  investigation  of  potential  sources  of  error. 
The  final  section  discusses  some  solutions  to  minimize  the 
effects   of    evaluator    bias. 

A.       AEPBCACH 

The  MCCFES  approach  to  evaluation  may  be  compared  with 
the  Professional  Review  (Accreditation)  Approach  fcrwarded 
by  House  [Ref-  1  ]•  It  is  an  evaluation  system  conceived 
within  the  Marine  Corps,  graded  by  Marines  and  using  stan- 
dards developed  by  Marines.  As  such,  it  closely  parallels 
the  Professional  Review  Approach.  In  this  approach,  a 
particular  profession  sets  standards  of  performance  for 
itself  and  conducts  internal  evaluations.  The  reasoning  for 
the  internal  evaluations  is  that  members  of  that  profession 
are    considered    experts  in  that   field. 
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In  chcosing  evaluators  to  perform  51CCRES  evaluations,  it 
is  desireable  that  evaluators  have  recently  served  success- 
fully in  a  billet  relating  to  the  function  they  are  to 
observe.  This  means,  for  example,  that  a  Rifle  Company 
evaluatcr  should  have  recently  served  successfully  as  a 
Rifle  Company  commander.  Successful  recent  billet  perform- 
ance increases  the  probability  that  evaluators  will  recog- 
nize adequate  mission  performance. 

B.   STRUCTURE 

The  MCCF.ES  evaluation  structure  is  a  four-tiered  hier- 
archy as  shewn  in  Figure  4.1.  of  particular  importance  to 
this  discussion  are  the  bottom  two  layers — the  Tactical 
Exercise  Controller  (TEC)  and  the  Evaluators.  it  is  here 
that    mission  performance   is    observed,    analyzed  and    reported. 


J       EVALUATION/EXERCISE   COMMANDER    "1 

i i 

I 

EVALUATION/EXERCISE    DIRECTOR! 


[    TACTICAL    EXERCISE    CONTROLLER! 


EVALUATORS 


Figure    4.1         MCCRES   Evaluation   Structure 
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1  •      Tactical  Exe rcise  C exit roller    (TEC) 

The  TEC  compiles  and  analyzes  the  results  cf  the 
evaluations  which  have  been  submitted  via  the  e valuator1 s 
data  sheets  and  submits  a  formal  report  to  the  Exercise 
Director.  Among  the  TEC's  duties  and  responsibilities  are 
determination  of  specific  Mission  Performance  Standards  to 
be  tested,  extensive  and  detailed  training  of  evaluatorsr 
development  and  control  of  intelligence  play  throughout  the 
problem,  and  organization  of  the  Tactical  Exercise  Control 
Group  to  plan  and  conduct  the  exercise.  The  TEC  relies  on 
the  evalautors  to  report  exercise  progress  and  mission 
performance  of  the  evaluated  units.  The  former  information 
is  received  primarily  via  radio  communication  while  the 
latter   arrives    in   the   form    cf    evaluator    data    sheets. 

2  •      Evaluators 

Evaluators    have  three    main  roles   in    the    MCCRZS: 

1.  Exercise  controllers  to  ensure  the  exer- 
cise  proceeds  as   planned. 

2-  Hl2i£§S  zo  resolve  disagreements  between 
exercise    and  aggressor  forces. 

3-  Performance  evaluators  to  observe  task 
performance  as  related  to  Mission  Performance 
Standards   being    graded. 

Ss  an  exercise  controller,  evaluators  work  as  an 
extension  of  the  will  of  the  TEC.  They  may  increase  or 
decrease  the  operational  tempo  of  the  problem  through  the 
use  cf  such  items  as  aggressor  forces,  intelligence  reports 
or  simulated  fires.  They  may  create  situations  which  require 
reaction  by  the  evaluated  unit  by  insertion  of  prescribed 
events  into  the  play  of  the  tactical  problem.  Action 
observed  at  this  level  is  provided  to  the  TEC  primarily  by 
radio  to  assist  the  TEC  in  determining  if  the  exercise  pace 
is   satisfactory. 
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As  umpires,  evaluators  are  tasked  with  resolution  of 
disagreements  which  may  occur  between  evaluated  units  and 
aggressor  forces.  For  example,  if  an  evaluated  unit  was 
ambushed  by  an  aggressor  force,  an  evaluator  would  make  a 
determination  as  to  the  outcome  of  the  ambush  and  assess 
casualties   accordingly. 

In  the  role  as  performance  evaluators,  evaluatcrs 
observe  unit  performance  of  prescribed  tasks  and  make  a 
determination  as  to  the  unit's  ability  to  satisfactorily 
carry  out  the  task.  These  determinations  are  recorded  as 
"YES",  "NO"  or  "NOT  APPLICABLE"  marks  on  the  evaiuatcr  data 
sheet.  A  mark  of  "YES"  denotes  that  all  facets  of  a  partic- 
ular requirement  were  met.  Conversely,  a  "NO"  mark  shews 
that  ail  portions  of  a  requirement  were  not  met.  "NOT 
APPLICABLE"  areas  are  those  net  tested  or  which  do  not  apply 
to   the   scenario    at   hand. 

Having  discussed  the  general  roles  of  the  evaiuatcr, 
two  topics  are  presented  to  help  explain  how  MCCRES  evalua- 
tors are  organized  and  what  measures  are  used  in  making  a 
determination  of  combat  readiness.  The  first,  Senior 
Evaluatcrs,  explains  the  duties  and  relationships  of  this 
MCCRES  member  to  the  rest  of  the  evaluators.  The  second. 
Mission  Performance  Standards,  looks  at  the  composition  of 
the    measures  used   in   conducting   the    MCCRES. 

a.      Senior    Evaluators 

Each  unit  evaluated  has  a  senior  evaluator  who 
conducts  a  post  exercise  wrap-up  and  compiles  the  data 
sheets  from  all  subordinate  evaluators.  At  this  wrap-up, 
resolution  of  each  "YES",  "NO"  and  "NOT  APPLICABLE"  rating 
is  made  for  each  requirement  tested.  This  resolution  of  the 
evaluator's  data  sheets  results  in  "YES",  "NO"  or  "NOT 
APPLICABLE"  ratings  for  each  requirement  as  it  pertains  to 
the    entire      unit.         The   senior      evaluator    provides      his   data 
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sheets  tc  the  TEC  fcr  compilation  and  further  use  by  the 
TEC.  An  assessment  cf  "COMBAT  READY"  or  "NOT  COMBAT  READY" 
for  the  entire  unit  is  also  also  passed  to  the  TEC  by  the 
senior   evaluator. 

The  senior  evaluator's  relationship  with  ether 
evaluatcrs  is  a  senicr-subordinate  type.  Senior  by  position 
and  generally  by  military  rank,  the  senior  evaluator  is  in 
charge  of  the  evaluation  team  and  is  responsible  fcr  evalu- 
ating the  performance  of  the  entire  unit  being  evaluated. 
The  senicr  evaluator  is  appointed  by  name  by  the  Exercise 
Director  (an  officer  senior  to  the  commander  of  the  organi- 
zation being  evaluated)  and  as  such,  maintains  an  indepen- 
dent relaticnship  to  the  organization  being  evaluated.  Other 
members  cf  the  evaluation  team,  subordinate  to  the  senior 
evaluatcr,  are  responsible  fcr  evaluating  the  subordinate 
units  (bcth  organic  and  attached)  and  other  organizational 
functions  (such  as  command  and  control  and  fire  support 
coordination)    of   the   overall   unit  baing   evaluated. 

b.      Mission    Performance   Standards 

Mission  Performance  Standards  (MPS*s)  are  stan- 
dards cf  task  performance  used  in  MCCRES.  Each  standard  is 
composed  of  various  tasks.  For  example,  the  MPS  Continuing 
Actions  By  Marines  is  composed  of  twelve  tasks  such  as 
Discipline,  Dispersion,  Security  and  Casualty  Handling. 
These  tasks  are  furtter  divided  into  conditions  and  require- 
ments. Conditions  specify  the  circumstances  under  which 
requirements  must  be  performed  and  provide  recommendations 
to  the  evaluator  concerning  time  and  space  limitations  which 
may  be  imposed  on  the  evaluated  unit.  Requirements  are 
specific  actions  which  must  be  performed  or  behaviors  which 
must  te  demonstrated  in  the  accomplishment  of  a  given  task. 
The  task  Discipline,  for  instance,  contains  nine  require- 
ments  ranging      from    Self   Discipline   and      Weapons    Maintenance 
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Discipline  tc  Hygenic  Discipline.  Requirements  which  may 
need  further  information  to  guide  evaluators  in  the  determi- 
nation cf  satisfactory  performance  are  provided  with  Key 
Indicators  (KI's)  of  performance.  KI's  are  an  attempt  to 
provide  an  objective  foundation  upon  which  to  base  an  evalu- 
ator's  judgement  of  satisfactory  requirement  performance. 
They  should  provide  specific,  measureable  actions  or  behav- 
iors which  must  be  present  for  the  requirement  tc  be 
successfully  completed. 

Consider  the  KI  for  the  requirement  Weapons 
Maintenance  Discipline.  "Marines  taka  care  to  clean  their 
weapons,  both  individual  and  crew  served,  daily.  Weapons  are 
safeguarded.  Care  of  weapons  enforced  by  leaders."  The  KI 
tells  what  is  to  be  done  (clean  weapons,  both  individual  and 
crew  served)  ,  when  it  is  to  be  done  (daily)  ,  who  does  it 
(Marines),  and  who  supervises  (leaders).  KI's  for  other 
requirements  provide  similar  types  of  information  to  make 
requirements  more  objectively    measureable    by   the   evaluator. 

C.       POTENTIAL    PROBLEMS 

This  section  discusses  the  areas  in  which  evaluators  may 
inject  bias  into  the  MCCRES.  The  discussion  is  presented  in 
three  parts:  Senior  evaluator  influence,  other  evaluator 
bias  and  MPS  problems.  Some  general  solutions  to  these  prob- 
lems are  suggested  here  with  more  specific  solutions 
presented    in  the   following    section. 

1  •      Senior    Evaluator    In  flu en ce 

The  senior  evaluator  can  inject  bias  in  two  major 
ways.  First,  as  the  senior  member  of  the  evaluation  team, 
he  or  she  sets  the  tone  for  the  other  evaluators.  If  the 
senior  evaluator  projects  a  hard-line,  "by  the  book" 
approach  toward  the  evaluation,  evaluators  may  tend  to  view 
task      requirements      with    little      flexibility.    On      the      ether 
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hand,  in  a  situation  where  the  senior  evaluator  projects  a 
less  rigorous  attitude  toward  the  evaluation,  evaluators  may 
tend  to  view  task  requirements  less  rigidly.  As  a  result  of 
evaluate!  perceptions  of  the  senior  evaluator's  wishes,  the 
evaluation    delivered   may   be    biased. 

The  second  major  way  in  which  the  senior  evaluator 
may  inject  bias  is  in  the  resclution  of  other  evaluator's 
ratings.  These  ratings  are  obtained  from  the  data  sheets  of 
the  other  evaluators.  The  senior  evaluator  depends  upon  the 
observations  made  by  the  other  evaluators  to  provide  data 
which  accurately  reflects  the  performance  of  the  entire 
unit.  Depending  on  the  senior  evaluator's  perceptions  of  the 
other  evaluators'  competence  and  on  his  own  perception  of 
successful  task  completion,  the  senior  evaluator's  data  for 
the  TEC  may  or  may  not  accurately  reflect  the  overall  ur.it*s 
abilities.  As  an  example,  suppose  an  infantry  battalion 
conducted  an  attack  on  an  aggressor  force  and  that  two  of 
the  companies  performed  extremely  well  while  one  company 
performed  poorly.  If,  in  the  senior  evaluator's  opinion, 
the  offending  company's  performanca  was  not  critical  to  the 
entire  unit's  mission  performance,  a  rating  of  "YES"  could 
te  delivered  for  the  battalion  for  the  task  "ATTACK"  as  it 
pertains  to  the  entire  unit.  [Ref.  26:p.I-C-8]  On  the  ether 
hand,  if  the  senior  evaluator  felt  the  one  company* s 
performance  was  such  that  it  negated  the  accomplishments  of 
the  other  two  companies,  a  rating  of  "NO"  could  conceivably 
te  returned  for  the  battalion  for  the  task  "ATTACK"  as  it 
pertains  to  the  entire  unit.  The  senior  evaluator  made  a 
decision  based  on  personal  judgement,  possibly  reflecting 
the    unit's   mission    performance   inaccurately. 

2 •      Other  Evaluator   Bia  ses 

The  evaluators  who  observe  task  performance  and 
report      to      the     senior        evaluator      are      presented      with     a 
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continuing   opportunity  to   inject    bias  into   the   MCCRSS.      The 

discussion    cf      the   areas    where      these  evaluators      may   inject 

bias      is   organized      in      two      groups:  errors     and      evaluator 
sources. 

a.      Errors 

Evaluator  bias  manifests  itself  as  any  deviation 
from  the  objective  'truth*  concerning  an  evaluated  unit's 
performance.  In  this  respect,  bias  may  be  regarded  as  an 
error  of  leniency,  strictness  or  hale  effect.  The  first  two 
errors  result  in  ratings  which  ara  respectively  too  "easy" 
or  too  "hard",  while  the  last  error  tends  to  cause  ratings 
to  grcup  arcund  one  value  en  the  rating  scale.  To  illus- 
trate, consider  an  evaluator  rating  the  requirement 
Equipment  Maintenance.  The  first  portion  of  the  KI  for  this 
requirement  states  "Vehicles,  generators,  etc.,  are  given 
close  attention  by  the  Marines  assigned  to  operate  them." 
[Eef.  26:p.II-A-6]  The  lenient  evaluator  may  consider  visual 
observation  each  feer  hours  constitutes  close  attention, 
while  a  strict  evaluator  considers  maintenance  conducted 
every  other  hour  as  an  indicator  of  close  attention.  If  a 
Marine  is  observed  by  these  two  evaluators  checking  his 
assigned  equipment  at  strict  four  hour  intervals  because 
that  is  what  the  operating  manual  calls  for,  he  will  receive 
a  different  rating  from  each  of  the  evaluators.  In  this 
case,  the  second  evaluator  has  injected  bias  by  committing 
the    error  of  strictness. 

As  an  illustration  of  halo  error,  suppose  an 
evaluator  is  rating  a  unit  on  a  task  which  contains  five 
requirements.  At  the  outset  of  the  observation  period,  the 
unit  was  particularly  outstanding  in  carrying  out  the  first 
requirement.  Based  upon  the  outstanding  performance  the 
evaluator  expects  similar  performance  for  the  other 
requirements  of      the    task.       Such  expectations      may   influence 
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the  evaluator  to  "see"  only  outstanding  performance. 
Mistakes  and  poor  performance  are  viewed  with  the  attitude 
that  "...they  really  know  better,  they  just  weren't  paying 
attention  today-..".  As  a  result  of  this  attitude,  a  "YES" 
rating  is  delivered  for  the  entire  task,  even  though  not  all 
requirements  were  successfully  completed.  This  evaluator  has 
committed  a  halo  errcr  since  the  rating  has  been  influenced 
by  the  outstanding  performance  of  only  one  requirement  of 
the  entire  task.  It  must  be  noted  that  this  error  can  also 
be  observed  in  the  opposite  sense,  that  is  a  particularly 
bad  observation  can  bias  the  evaluator  to  view  an  entire 
task    unfavorably. 

b.      Evaluation   Sources 

In  the  previous  discussion  of  the  three  main 
sources  cf  evaluaticn--superior,  peer  and  disinterested 
party — it  was  shown  that  the  first  two  sources  demonstrate 
fairly  ccmparabie  errcr  introduction  but  may  vary  greatly  in 
perceptions  cf  task  performance.  This  difference  in  percep- 
tion is  related  to  the  dimensionality  of  the  task  being 
evaluated.  In  the  context  of  MCCRES  this  means  that  supe- 
riors may  net  perceive  task  performance  in  the  same  way  as 
peers.  The  last  evaluation  source,  the  disinterested  party, 
brings  with  it  the  potential  problem  of  not  understanding 
the    process    being  graded. 

Many  of  the  potential  problems  associated  with 
various  evaluation  sources  are  diminished  by  two  MCCRES 
stipulations  concerning   evaluators.  The    first    stipulation 

is  that  evaluators  should  have  recently  served  a  successful 
tour  in  a  billet  related  to  the  one  they  are  evaluating.  k 
key  word  in  this  stipulation  is  recently.  Since  billets  in 
the  Marine  Corps  have  ranks  associated  with  them,  the 
differential  dimensionality  of  senior  and  peer  evaluators  is 
limited    by   ensuring    evaluators  have    recently    filled   a   billet 
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similar  tc  the  one  they  are  evaluating.  In  other  words,  an 
evaluator  who  has  recently  served  in  a  billet  similar  tc  the 
one  he  is  evaluating  is  more  likely  to  recognize  those  task 
dimensions  which  indicate  successful  task  performance  than 
an   evaluator  who   has   not   recently  held   such   a   position. 

Besides  the  problems  associated  with  differen- 
tial dimensionality  between  evaluation  sources,  social 
interaction  between  sources  and  the  evaluated  unit  can  be 
problematic.  Both  seniors  and  peers  within  an  organization 
tend  to  interact  in  formal  as  well  as  informal  ways.  This 
informal  or  social  interaction  may  be  carried  into  the  eval- 
uation as  a  bias.  The  second  stipulation  states  "...it  is 
desireable  that  evaluators  be  obtained  from  adjacent 
commands  not  directly  related  to  the  organization  being 
evaluated."  [  Ref.  26:p. I-C-9]  This  may  result  in  a  reduc- 
tion cf  bias  created  by  social  interaction.  This  reduction 
is  due  to  decreased  daily  interaction  between  members  of 
adjacent  units  as  compared  to  daily  interactions  among 
members    cf    a  single    ur.it. 

■3  •      Mission    Performance    Standards 

All  of  the  evaluation  sources  have  one  thing  in 
common:  they  use  the  Mission  Performance  Standards  to  eval- 
uate unit  combat  readiness.  A  potential  problem  associated 
with  the  MPS' s  is  their  subjectivity.  This  subjectivity 
permits  evaluator  interpretation  of  standards  which  nay 
result    in   biased  evaluations. 

To  determine  the  extent  of  the  MPS's  subjectivity, 
the  requirements  fcr  the  MPS's  Continuing  Actions  By 
Marines,  Command  And  Control  and  Fire  Support  Coordination 
were      examined.  The     criterion        used      to      determine      the 

subjectivity  of  a  requirement  was  the  ability  of  the 
requirement  to  be  quantified.  If  the  requirement  was 
expressed   in  terms    which   are   physically    measureable,    such  as 
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units  of  time  or  distance,  then  it  was  considered  objective. 
Requirements  containing  phrases  which  require  interpretation 
by  the  evaluatorr  such  as  "...close  attention...",  were 
considered  subjective.  The  meaning  of  these  requirements  can 
depend  upon  theevaluator' s  interpretation  of  the  require- 
ment's  wording. 

Cf  the  243  requirements  for  the  above  MPS's,  15  were 
found  to  be  susceptible  to  evaluator  interpretation.  This  is 
approximately  6.2  percent  of  the  requirements  for  these 
three  MPS's.  These  15  requirements  contain  phrases  such  as 
"...close  attention..."  or  "...processed  with  speed..."  to 
describe  satisfactory  requirement  performance.  Without  clear 
guidance  as  to  what  constitutes  "close  attention"  or 
processing  "with  speed",  different  evaluators  may  interpret 
the  requirement  to  have  different  meanings.  This  difference 
in  interpretation  means  that  two  evaluators  observing  a 
particular  requirement  being  performed  could  return 
different  ratings  of  requirement  performance,  depending  on 
how  the  requirement  is  interpreted.  For  each  of  the  15 
requirements,  the  requirement  number  and  the  subjective 
phrase   contained   in    the   requirement   is   listed   in    Tabls    II. 

D.       POTENTIAL    PROBLEMS    PERCEIVED    BY    FIELD    USERS 

Six  Marine  officers  attending  the  Naval  Postgraduate 
School  were  interviewed  to  gain  an  insight  into  potential 
MCCRES  problems  as  perceived  by  users  in  the  field.  The  six 
officers  ranged  in  grade  from  0-2  to  0-4  and  represented 
MOS's  0302  (Infantry  Officer)  1302  (Engineer  Officer)  7562 
(Pilct  HMM  CH-46)  and  7587  (Airborne  Radar  Intercept 
Officer,        F4N/J/S) .  The      interview      consisted      of      three 

questions: 

1.  Do   you      feel    that      an   evaluator      can   affect     a 
MCCRES    evaluation   through   personal    bias? 

2.  How    is    this    bias   input? 
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3.      In    what    areas   do   you    feel    oias    is   most    likely   to 

cccur? 
The  results  of  these  interviews  demonstrated  that  there  was 
close  agreement  en  each  of  the  questions  across  both  MOS  and 
grade.  All  interviewees  felt  that  an  evaluator  could  affect 
a  MCCEES  evaluation  through  personal  bias.  This  bias  was 
seen  as  being  input  through  evaluator  interpretation  of 
performance  criteria.  These  criteria  take  the  form  of  task 
requirements.  Responses  to  the  last  question  indicate  field 
users  felt  bias  is  most  likely  to  occur  in  those  areas  to 
which  numerical  measures  are  not  easily  attached.  They  felt 
areas  which  lend  themselves  to  quantifiable  measurement  are 
less  likely  to  contain  evaluator  bias  than  non-quantifiable 
area  s . 

TABLE    II 
HFS    Requirements   Susceptible   to   Evaluator   Bias 


Requirement    Number 

""      tatttt.i 

2A.  1.1. 4 

2A.  1.1.7 
2A.  1.1.8 
2A.  1.3. 6 

2A.  1.11.  14 
2A . 2.7  .  2 
2A.2.8.2 

2A.2.9.  5 
2A.2.9. 6 

2A.  2.10.5 
2A.3.4. 5 
2A.3.4.7 
2A.3.5.3 
2A.3.5.7 


Subjective   Phrase 
"^close    attention17 
"orderly   and  organized 
fashion" 

"exhibit    restraint" 
"light    use   to   a    minimum" 
"COHSEC    material    safe- 
guarded" 

"processed   with   speed" 
"provided   with   security" 
"safeguards    classified 
material" 

"neat    and   orderly" 
"dispersed   to  reduce 
vulnerability" 
"dispersed" 
"closely    monitors" 
"timely   manner" 
"accurate    plots" 
"closely   monitors" 


Comparison  of  potential  problems  with  HCCHES  as 
perceived  by  the  sample  of  field  users  to  the  potential 
problems  outlined  in  the  previous  section  shows  that  the 
field  users'  perceptions  are  a  subset  of  the  potential  prob- 
lems   discovered    through   analysis   of   the    MCCRES. 
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E.       RECOMMENDED    SOLUTIONS 

The  problems  discussed  in  the  previous  two  sections 
demonstrate  the  variety  of  ways  in  which  an  evaluator  may 
introduce  bias  into  a  MCCRES.  In  order  to  minimize  bias 
input,  three  possible  solutions  to  the  bias  problem  are 
forwarded.  These  solutions  are  evaluator  training,  evaluator 
testing   and    quantification    of   subjective   MPS    requirements. 

1  •      Evaluator  Training 

As  previously  noted,  evaluator  training  has  proved 
to  be  an  effective  tool  in  reduction  of  evaluator  error. 
Bernardin  [Ref.  23]  showed  that  evaluators  receiving  compre- 
hensive training  shew  greater  error  reduction  results  than 
evaluatcrs  receiving  limited  training.  Both  of  these  groups 
show  less  error  than  evaluators  who  have  received  no 
training. 

Current  MCCFES  standards  -ask  the  TEC  with 
conducting  extensive  and  detailed  training  of  evaluators.  In 
the  experience  of  several  officers  attending  the  Naval 
Fostgraduate  School,  who  were  questioned  concerning  evalu- 
ator training,  this  training  is  geared  toward  educating  the 
evaluator  on  the  exercise  scenario  with  no  specific  mention 
of  the  errors  which  evaluators  typically  commit.  By  making 
MCCHES  evaluators  aware  of  the  errors  typically  committed  by 
evaluators,  the  MCCEES  evaluators  are  less  likely  to  commit 
these  errors,  reducing  biased  input.  An  evaluator  training 
package  addressing  both  scenario  development  and  possible 
evaluator  error  should  be  created  to  more  fully  exploit  the 
potential  of  comprehensive  evaluator  training  outlined  by 
Eernardin  [Bef.    23]. 

Another  aspect  of  evaluator  training  is  ensuring 
potential  evaluators  are  well-versed  in  the  areas  they  are 
chosen   to   evaluate.      Choosing    knowledgeable   evaluators   tends 
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to  increase  the  probability  that  those  factors  which  indi- 
cate successful  task  performance  are  considered  during  the 
evaluation. 

Cne  method  to  ensure  trained,  knowledgeable  evalua- 
tors  for  MCCRES  evaluations  is  formation  of  a  formal  MCCRES 
evaluation  team.  By  choosing  team  members  who  have  demons- 
trated proficiency  in  their  MOS's  and  keeping  them  current 
in  both  their  MOS's  and  evaluation  techniques  through 
training,    a   skilled    cadre  of   evaluators    can   be   assembled. 

Seme  of  the  advantages  of  forming  a  formal  MCCRES 
evaluation  team  are  minimization  of  evaluator  training 
costs,  minimization  of  social  interaction  with  evaluated 
units  and  a  more  standardized  evaluation  base.  Evaluator 
training  costs  are  irinimized  since  the  same  evaluators  are 
frequently  used.  Though  training  effects  diminish  rapidly 
with  time,  retraining  for  each  successive  evaluation  could 
demonstrate  a  learning  curve,  reducing  costs  ever  time. 
Social  interaction  is  minimized  due  to  lower  daily  contact 
with  evaluators,  as  opposed  to  the  interaction  which  occurs 
among  adjacent  commands.  The  last  factor,  standardization  of 
the  evaluation  base,  results  from  the  continuity  of  the 
formal   evaluation   team. 

A  MCCRES  evaluation  team  could  be  composed  of 
personnel  from  units  such  as  Division  Schools,  or  it  could 
reside  outside  the  active  duty  forces  at  a  Reserve  unit, 
since  the  MCCRES  is  to  evaluate  both  active  and  reserve 
forces.  Having  reserves  evaluate  MCCRES  would  also  offer  the 
additional  benefit  cf  keeping  the  reserve  up  to  date  and 
strengthening  the  tie  between  active  and  reserve  forces  in 
the    Marice   Corps. 

2  •      Evaluator  Testing 

Evaluator  testing  can  be  seen  as  a  method  of  both 
controlling     and      controlling      for      evaluator      bias.    In      the 
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former  case,  a  test  can  be  constructed  which  would  indicate 
the  areas  in  which  a  prospective  evaluator  demonstrates 
bias.  By  testing  a  number  of  these  prospective  evaluators, 
those  who  demonstrate  little  or  no  bias  could  be  chosen  to 
conduct  MCCRES  evaluations,  thereby  minimizing  the  likeli- 
hood of  evaluator  bias  input.  For  instance,  consider  a  test 
in  which  evaluatcrs  are  graded  according  to  their  agreement 
with  an  answer  key.  Further,  suppose  the  answer  key  is 
composed  cf  the  pooled  answers  of  a  group  of  "unbiased" 
evaluators.  As        suggested        by  Wiley        and        Jenkins 

[Ref.  24:p.217],  evaluator  agreement  with  the  key  can  be 
used  to  predict  the  likelihood  of  evaluator  bias.  These 
evaluators  showing  close  agreement  with  the  key  of  "unbi- 
ased"  answers  can   be    chosen    to   perform   evaluations. 

The  same  test,  analyzed  differently,  can  be  used  tc 
control  for  evaluator  bias.  For  instance,  the  results  of  the 
test  are  analyzed  tc  discover  in  which  areas  an  evaluator's 
biases  exist.  From  this  analysis  a  "bias  profile"  could  be 
constructed  which  could  allow  evaluation  results  tc  be 
"standardized".  For  example,  assume  a  MCCRES  evaluator's 
bias  profile  showed  significant  deviation  toward  strictness 
in  the  area  of  discipline.  During  the  conduct  of  a  MCCRES 
evaluation  a  senior  evaluator  notes  this  evaluator's  data 
sheet  has  a  "NO"  rating  for  many  of  the  requirements  cf  the 
task  DISCIPLINE.  The  senior  evaluator,  knowing  that  this 
evaluator  tends  to  be  particularly  strict  in  evaluating 
discipline,  may  wish  to  obtain  additional  performance  infor- 
mation concerning  the  unit  evaluated,  since  the  evaluator's 
ratings  may  not  accurately  reflect  the  unit's  actual 
performance. 

3  .      Cuantif  icaticn  of   UBS1  s 

The  last  method  of  controlling  evaluator  bias  is 
quantification        of        subjective        MPS        requirements.      This 
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quantification,  as  Scott  [Hef.  20]  suggests,  reduces  the 
©valuator's  task  from  interpreting  MPS  requirements  and 
comparing  task  performance  with  this  interpretation  to 
reporting  whether  task  performance  meets  the  requirements. 
For  example,  instead  cf  trying  to  decide  how  fast  the  phrase 
"...  process  with  speed..."  is,  reporting  whether  the  unit 
was  able  tc  "...process  within  two  hours..."  is  less  open  to 
interpretation.  Ihe  more  concrete  the  requirement,  the  less 
evaluator  interpretation  that  will  take  place  in  grading, 
resulting  ir  reduced  evaluator  bias.  Some  of  the  quantifica- 
tions may  be  less  concrete  than  others.  Some  requirements 
may  be  constructed  in  terms  of  ranges  of  acceptable  perform- 
ance for  differing  tactical  scenarios.  Still,  the  ranges 
serve  to  bound  the  amount  of  interpretation  required  by  the 
evaluator. 

F.  CCHCI0SIONS 

In  the  introduction  of  this  paper  two  questions  are 
posed.  The  first  asks  if  factors  of  the  MCCHSS  evaluation 
which  are  subject  to  evaluator  bias  can  be  identified,  and 
the  second  asks  how  these  factors  can  be  controlled  or 
controlled  for.  It  has  been  shown  that  areas  in  which  evalu- 
ators  may  bias  the  MCCHES  can  be  identified  and  comprise 
three  basic  areas:  senior  evaluator  influence,  other  evalu- 
ator   bias   and  MPS   interpretation. 

As  for  methods  cf  controlling  or  controlling  for  these 
factors,  three  techniques  were  forwarded:  evaluator 
training,  evaluator  testing  and  quantification  of  subjective 
MPS  requirements.  Each  of  these  techniques  has  potential  for 
controlling   bias. 

G.  RECCHHENDATIONS  FOR  FUTURE  RESEARCH 

Discussion  of  the  proposed  solutions  to  the  problem  of 
evaluator      fcias    did      not      address   the      cost   to   implement   the 
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solutions.  A  study  cf  benefits  and  costs  for  each  cf  the 
solutions  would  provide  additional  information  as  to  the 
feasibility  of  the  solutions.  In  addition,  a  detailed  study 
cf  the  proposed  solutions  would  be  likely  to  point  out 
several  methods  of  implementation  for  each,  possibly 
revealinc  still  other  solutions  not  addressed  in  this 
thesis. 
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